Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable full passthrough of vllm engine args, with backwards compatibi… #2639

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

siddvenk
Copy link
Contributor

…lity for existing differences with engine agnostic configs

Description

This PR addresses a common pain point with using vllm. We currently restrict the set of engine args that can be passed based on what we expose through VllmRbProperties. This change does the following:

  • enable pass through args via Pydantic extra configs
  • remove configs from VllmRbProperties that are exactly the same as the vllm engine args in both name and default value
  • validate specific configs that we have different engine agnostic names for

Raising this in draft if people want to start looking and providing feedback. I have the following TODOs still:

  • Add unit tests for the properties class to test this change in behavior
  • Add documentation updates to instruct customers on how to use this change correctly

I plan on making a similar change for LMI-Dist. I'm starting with VLLM first to keep the change contained, and gather feedback suggestions which I'll take forward for LMI-Dist.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Checklist:

  • Have you manually built the docker image and verify the change?
  • Have you run related tests? Check how to set up the test environment here; One example would be pytest tests.py -k "TestCorrectnessLmiDist" -m "lmi_dist"
  • Have you added tests that prove your fix is effective or that this feature works?
  • Has code been commented, particularly in hard-to-understand areas?
  • Have you made corresponding changes to the documentation?

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

  • Test A
    Logs for Test A

  • Test B
    Logs for Test B

@siddvenk siddvenk force-pushed the vllm-engine-args-passthrough branch 6 times, most recently from 768e4c1 to a8f0e0f Compare December 16, 2024 23:25
@siddvenk siddvenk force-pushed the vllm-engine-args-passthrough branch 4 times, most recently from 1c0bc87 to b92bc80 Compare December 18, 2024 22:08
…lity for existing differences with engine agnostic configs
@siddvenk siddvenk force-pushed the vllm-engine-args-passthrough branch 6 times, most recently from 1ecdbcf to 747018c Compare December 20, 2024 18:48
@siddvenk siddvenk force-pushed the vllm-engine-args-passthrough branch 11 times, most recently from 7171eb2 to 9aa3165 Compare December 21, 2024 00:03
@siddvenk siddvenk force-pushed the vllm-engine-args-passthrough branch 2 times, most recently from e1b9854 to 541bd63 Compare December 21, 2024 00:57
@siddvenk siddvenk force-pushed the vllm-engine-args-passthrough branch from 541bd63 to 59bd468 Compare December 21, 2024 01:13
@siddvenk siddvenk marked this pull request as ready for review December 21, 2024 01:18
@siddvenk siddvenk requested review from zachgk and a team as code owners December 21, 2024 01:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant